Ten Years of Rapid Development

Over the past ten years, Tcl has been a key development tool for a range of collaborative systems, ranging from academic research prototypes to large-scale use in commercial web conferencing products. This paper examines how this use of Tcl evolved, highlighting some lessons for developers of growing systems, and focusing on how well Tcl can support traditional software engineering practice. Tcl was also made a focus of technical due diligence during a company acquisition, and many of the positions offered in its support may prove useful to others facing management pressure over their choice of development tool. 1. If Development was so Rapid... For the past ten years, I’ve been involved in the development of a series of collaborative systems built using Tcl. These systems encompassed the spectrum from a Tcl extension to aid in academic research prototyping, to several large, robust desktop and web-based commercial products. Like so much Tcl development work, very small and focused development teams built these various systems, even as some of the software grew larger and more complex. One constant theme was experimentation and reinvention; a wide variety of systems evolved over the ten-year period. It’s safe to say that some of the abrupt corner-turns, particularly during the dot-com frenzy, necessitated a constant stream of rapid redevelopment and change. This paper will cover several phases of the work. It began around 1992 with GroupKit, a university-developed Tcl extension used for rapid prototyping of collaborative interfaces and experimentation with underlying software architectures. A followon application in 1996, TeamRooms, was a more robust desktop collaborative environment combining a range of tools. TeamRooms was later spun-off into a company I founded, TeamWave Software Ltd., and the software was further developed as TeamWave Workplace. TeamWave moved to web-browserbased products in 1999, applying the technology to virtual communities, music sharing, e-learning, and web conferencing/meetings. The company was eventually acquired in late 2000; our use of Tcl was a key issue during the technical due diligence phase leading up to the acquisition. Throughout this paper, I’ll highlight many of the issues that we used to pitch Tcl as a significant strength, rather than a liability for the acquirer. While some of the issues are familiar (e.g. rapid development time, easy integration), others are less obvious, such as arguments surrounding performance and scalability. These can depend heavily on Tcl development practices, and effectively using Tcl in support of sound software engineering practices. While best practices aren’t always obvious, and there are some issues with practical use in organizations, all the benefits certainly made it an excellent choice at every phase of our development work. 1.1. Experience papers Earlier years of this conference have brought a variety of experience papers, most notably Don Libes’ paper reflecting on seven years of evolution of the Expect extension [5], and Tom Phelps’ paper on building the TkMan application [9]. Through these and other papers, we’ve learned practical techniques for common problems, more obscure issues that creep up during longer-term deployment, and examined the effect of aesthetic language issues and design choices. Through war stories and experiences, we can avoid pitfalls and discover new practices to aid in our own development, and understand better some of the factors affecting core language evolution. This paper differs somewhat from previous experience papers. It covers a wider range of systems, examines software engineering practices with Tcl as code and teams start to grow, and addresses some of the “soft” concerns such as organizational pressures as they impact the choice of development tools and practices. 1.2. Collaborative Systems The application domain I’ve been working in is collaborative systems, a rather broad area that covers a range of things from chat systems, to email, to bulletin boards, to video conferencing and more. Other names often applied to the same domain include “groupware” and “conferencing”. These systems are shared by several people over a network, and tend to be classified according to whether they allow people to work together at the same time such as in a meeting (“real-time” or “synchronous”) or if they are to support people working together over a longer period of time at different times (“asynchronous”). Most of my work has been in more visually oriented systems, often focused around visual surfaces like whiteboards or slide shows. These tend to be highly interactive areas, where gestures, annotations, object manipulation, and other small actions must be quickly communicated to everyone sharing the system. Developing these systems is interesting, because not only do they involve many complex technical issues (network communications, data storage, security, etc.) but because people are actually using these tools to try to communicate with each other. That means that user interfaces (and even social context) become very important. It is no surprise then that collaborative systems share many characteristics with other computational, client-server, networked or user interface applications commonly developed with Tcl/Tk. 2. GroupKit: Prototyping Interfaces GroupKit [10,13] was an open source Tcl/Tk extension developed at the University of Calgary, providing common collaboration facilities to aid in the prototyping and evaluation of groupware applications and architectures. It was used by many students and researchers at various institutions. Figure 1 shows some examples of the types of tools prototyped with GroupKit. GroupKit had actually started life as a C++ class library, based on the InterViews GUI library. The move to Tcl/Tk (precipitated by some eager experimentation immediately after the first release of Tcl-DP) took about one month and yielded some benefits over the C++ version by now very familiar to most Tcl/Tk users: • The easier learning curve of Tcl/Tk made the system accessible to more people; ramp-up time for those unfamiliar with Tcl was measured in hours or days, not weeks, making GroupKit accessible to coursework for students. • The rich Tk widget set, especially the canvas and text widgets, enabled more sophisticated interfaces to be prototyped more easily and quickly. • Code size, development time, and debugging time shrunk by an order of magnitude for typical applications, which was critical for the target audience. • We were later able to develop Windows and Macintosh ports of GroupKit, supplementing the original Unix version. As a result, the system gained significantly more widespread use in the research community, more interesting systems were built, we could add many more features, make it flexible enough for a wider range of uses, and do more of our own experimentation. In many ways, this was a good example of the traditional use of Tcl/Tk, making prototypes and small applications, developed by very small numbers of often-novice developers. Using Tcl/Tk helped bring down the scale of larger problems. It let developers think at a higher level about the problem, and reduce the amount of code needed to express the solution. As a result, problems were solvable more quickly and easily by individual developers. Figure 1. Examples of interfaces developed with GroupKit. 3. TeamRooms: A Larger Application TeamRooms [11], and later the commercial TeamWave Workplace, was a full-fledged collaborative application, rather than a prototyping environment. It provided members of a group a set of electronic rooms where they could share work, both in realtime and asynchronously. The rooms combined a variety of different shared tools. Figure 2 shows an example. These systems relied on a central server, along with a desktop application that ran on each client’s machine. Both client and server ran on Mac, Windows and several Unix platforms. Internally, the systems were fairly standard Tcl/Tk applications. A novel feature was that the client used a number of different Tcl interpreters to juggle the different tools in the environment, allowing new tools to be added, yet from a programmer’s point of view appear isolated from the rest. The application, though considerably larger and more complex than earlier work in GroupKit, was still reasonably sized (<50k loc), and was built primarily by 2-3 developers. Development challenges primarily surrounded then-bleeding-edge features in Tcl, such as the cross-platform versions of Tk and the core’s new I/O model and socket support. Performance requirements dictated a modest amount of spot recoding in C and other isolated optimizations, all easily identified using standard profiling. TeamWave Workplace was again typical of many more mature Tcl/Tk applications, becoming more robust and full-featured with each version, but still developed on a fairly small scale. The relatively small code base and size of the developer team allowed ad hoc development practices to suffice. The coding style (e.g. documentation, naming conventions) was fairly loose, intermodule dependencies were high, and testing was largely manual. This looseness was conducive to the rapid changes that occurred, especially in early versions of the commercial system, when many new features were added to meet customer needs. But as we’ll see soon, the ad hoc development practices started causing problems. Figure 2. Example of TeamWave Workplace interface. 4. Moving to the Web By late 1998 it became clear that the TeamWave application had to be retargeted to run inside a web browser, rather than as a double-clickable desktop application. For many of our target customers in education and virtual communities, the need to download software and run another application was too large a barrier for new users, and an impediment to regular use. While modest angel funding (by dot-com standards!) had allowed us to grow our development team to 5-10 people, trying to do a complete rewrite (e.g. as a conventional Java applet) was both too large a project and too risky given the state of client-side Java at the time. We wanted a solution that would preserve as much of our code base as possible, and enable us to continue our rapid development using Tcl if we could.